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[57] ABSTRACT 

This invention relates to a method and apparatus for a 
system having a multisensor optical module with a large 
aperture angle, 180 degrees or more possible (360 de- 
grees- around), and an electronic interface which out- 
puts two video or digital images of the scene. The first 
image is a high resolution representation of the entire 
field of view reconstructed from the partial views of 
each sensor. The second image is a high resolution rep- 
resentation of a selected part of the scene (window), the 
dimension and position of which can be interactively 
commanded through a communication port The size 
and location of the window may be changed at every 
frame (typically every l/30th of a second). The win- 
dow field of view can cover the field of view of several 
adjacent sensors while the resulting image will remain 
continuous and undistorted. An arbitrary number of 
windows can be obtained with a unique optical sensor 
by adding electronic modules to the interface apparatus. 
The system is applicable, for example, in professional 
video and television, in industrial machine vision, in 
image processing, in surveillance systems, in teleguided 
and tele-operated systems and in many other fields. 

11 Claims, 4 Drawing Sheets 
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stead of treating all pixels of all sensors as part of one 
VIDEO IMAGING SYSTEM WITH INTERACTIVE video image, so that the integration is in fact made by 
WINDOWING CAPABILITY the observer's eyes; 

Selecting at a video rate the "region of interest" (win- 
BACKGROUND OF THE INVENTION 5 dow, monitor) pixels from different sensors and arrang- 
The present invention relates to an improved appara- kg them to give a high resolution continuous image; 
tus and method for providing high resolution images of Changing at every new frame (illustratively every 
a scene, and for obtaining simultaneously a wide angle l/30th of a second) the window parameters (position, 
image and a "zoomed" image of a particular region of length, width); and 

the scene ("window"), with both images having high 10 Defining an unlimited number of windows by adding 
resolution. the necessary number of modular electronic cards in the 

Conventional systems used today for similar purposes electronic interface. 

9 TS£t££ mounted on . mecl»ni«l orient.. OBJECTS OF THE INVENTION 

tion system; or a 15 It is a general object of the present invention to pro- 

2) Video camera with an electronic interface system vide a new and improved method and apparatus com- 
which displays only part of the image seen by the prising an optical sensor and electronic interface for 
camera. outputting two video (or digital) images of a wide angle 

These prior art systems have the respective disadvan- scene. The first video output gives an image of the 
tages of: 20 entire scene, with a possible aperture angle of 180 de- 

1) Low speed of orientation and no environmental grees or more (360 degrees around). The second video 
monitoring, i.e., only the region aimed at is seen. output gives an image of a particular region of the scene 
Also, mechanical systems are much less reliable (window). The position and size of the window are 
than electronic systems; communicated to the system by means of a standard 

2) Low resolution of the image in partial view. In 25 ^ and may ^ changed at every new 
order to obtain high resolution images large frame, typically thirty times per second. 

CCD. arrays can be used, but the price of these of ^ principal advantages of this invention 

arrays increases very rapidly with increased resolu- m , 

tion, and the time needed for reading these arrays 1} ^ h ^^on of very ^ Mgle scene, 
increases proportionally as the sqtiared yalueofthe 30 rcsolution i^ge 0 f the '^window", 

image resolution, thus lowering Ae image rate. 3) ffi h fate ofchazige ofwiBdow parameters (size and 
Furthermore, wide angle views suffer from focus- no&'tio 1 

ing and distortion problems since ^imagespheri- > & w ^ ^ guch 

cal surface does not fit the plane sensor surface. r rr% 

Prior United States patents in related fields whkh 35 *" ^^L>- "ray* 
make use of several sensors include: g J?** 5 » te dependent of the image resolution. 

Combining several video signals into one, giving the $ £o moving parts, allowing rugged comtrucUon 
possibility of displaying one of them (U.S. Pat. Nos> D Possibility ofopemng additional windows by adding 
4,777,526 and 4,814,869); electronic modules. 

Selecting a view from a number of cameras and pro- 40 BRIEF SUMMARY OF THE INVENTION 
cessing each one in a specific way (U.S. Pat. Nos. 

4,779,134, 4,831,438, 4,462,045, 4,577,344, 4,630,1 10 and ^ preferred embodiment of the present invention 
4,943,864)* comprises two primary parts: 

' Making* a stereoscopic view by use of two cameras ] > J}" 5 °P tical module 
(or volume sensing) (U.S. Pat Nos. 4,739,400, 45 2) electromc interface 

4 395 731 4 559,555 and 4,709,265)- Tne °P tlca ^ module is comprised of a number N of 

'selecting'region of interest from one camera (U.S. Kght sensitive arrays (illustratively, CC D. arrays), 
Pat No. 4,713,685); disposed on an hemisphere-like surface, with each array 

Moving* the area 1 viewed by electronic means in a having its own lens system. Every sensor element cov- 
"vidicon" (U.S. Pat. No. 4,740,839); 50 crs part of the total field of view so that the entire field 

Adding some necessary data to video signal (U.S. °* v« w « covered by all of the sensors elements, with 
Pat. No. 4,922,339); ^° me necessary overlapping. Every sensor element is 

Projecting a 360 degree field of view on one spherical connected to the electronic interface, 
surface and exainining it by the use of one rotating The electronic interface receives data from all of the 
sensor (U.S. Pat No. 3,588,517); 55 s«nscr elements and makes an integration of such data. 

Processing four camera views so that they will give Date from all sensors is collected and digitized in paral- 
one larger view when displayed on four adjacent moni- lei by synchronous scanning. Then, the integration is 
tors (U.S. Pat. No. 4,777,942). made °y selecting, at every stage of scanning, the par- 

None of these prior art teachings contains the novel ticuiar data that is relevant for the current pixel of the 
and highly advantageous features of the invention de- 60 considered video (digital) image. As a result, there is no 
scribed here, which are: need to store the total amount of information contained 

Collecting all pixel data of a determined field of view in sensor elements, and a low cost implementation is 
on a number of sensors, using a specific arrangement of possible. 

elements; accordance with the improved method and appa- 

Treating the pixel data at a video rate (typically 10 65 ratus, two video (digital) images are produced, namely: 
Mhz) as part of a global total scene. In U.S. Pat. No. 1) "Monitor Image": This image gives a continuous 
4,777,942 one global image is built from four different representation of the entire field of view. Several repre- 
sensors, but this is done by monitor juxtaposition, in* sentation models are possible, since the basic shape of 
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field of view is spherical while the "Monitor Image" is module 10 comprises a light sensitive array formed of a 

a plane representation. For example, if the field of view sensor element 24 and a lens 26. 

has a 180 degree aperture angle, a polar representation Every sensor element 24, whether on the optical 

can be chosen. module 10 in the FIG. 2 arrangement or in the FIG. 3 
2) "Window Image": This image gives a continuous 5 arrangement, is connected to the electronic interface 12 

representation of a selected portion of the field of view. so the Utter receives all data from ill the sensor ele- 

This portion can cover a number of sensors but the ments to make an integration, as further described 

transition from one sensor to an adjacent one is not **low with respect to the illustrative arrangement of 

perceptible on the "Window Image". The size and posi- FI G - 

don of that "Window" (window parameters) is trans. 10 A* illustrative electronic interface schematic block 

mined to the system via a dedicated standard communi- diagram is shown in FIG. 4. A detailed description of all 

cation port, from a host device like a joystick or a host modules of this electronic interface embodiment fol- 

computer. A new set of window parameters can be lows: 

down loaded at every new image frame, typically thirty Sweep address generator 30 

times per second. ~. . . -_t * w 

By adding a desired number of electronic modules in ™* «™« S OTCr !£' *} * rate, a signal for 

parallel, several window images can be produced using ^ ^w* 

IT, • ' ^„^ al —^..i. scanning address that will be used as entry for tables in 

an unique optical module. ^ £p £ OM mcmories . ne 5wecp generator 

BRIEF DESCRIPTION OF THE DRAWINGS 20 30 can be of any of the various types known in this art 

FIG. 1 is a schematic description of the system com- for Providing this function. 

ponents and connections, in accordance with a pre- A/D converter 32 and multiplexers 34 

feiredem^ent of the invention; An analog to digital converter 32 (A/D) is corrected 

FIG. 2 is an illustrative arrangemen of tfie invention ^ * Q ^ q{ ^ ^ from ^ 

having 15 sensors and lenses, giving a 180 degrees aper- 8cnsor ^ digitUc / by N g^og t0 digital converters 32 
* * r,». • (one per sensor), and is entered into two N to 1 multi- 
HG. 3 is another illustrative arrangement of the m- ^£34, ^ conversion is made at standard video 

vention having 8 sensors and lenses, giving a ±20 and rFlash/A/tr) . Th c a/D converters and multi- 

360 s around field of view, applicable, for example, in ^ lcxeK 34 ^ ^ be of any type known in this art for 

vehicle teleguiding; and providing these functions. 
FIG. 4 is a general block diagram of an illustrative 

electronic interface. EPROM 1 and EPROM 2 

DETAILED DESCRIPTION OF A PREFERRED The illustrative embodiment of the preferred inven- 
EMBODIMENT OF THE INVENTION 35 tivc system arrangement utilizes 2 EPROMS: 

The first EPROM 36 contains a sampling set of sen- 

As shown in FIG. 1, the preferred embodiment of the ^ selection numbers and coordinates, which are pre- 
ventive system is divided in two parts, namely, the determined for giving a good "monitor" image. At 
optical module 10 and the electronic interface 12. cver y scanned address, the EPROM outputs a "select 

The optical module 10 is comprised of a number N of 40 sensor" number which indicates from which sensor 
light sensitive arrays 14 disposed on an hemisphere like element the data is to be taken now, and a "Monitor 
support surface 16. Frame Buffer" address which indicates at which place 

The output from the sensor elements of the optical u the monitor image the data will be stored. The 
module 10 is provided to the electronic interface 12. EPROM table is built so that if there are N sensor ele- 
The latter is connected to two video displays 18 and 20, 45 me nts, 1 every N pixel of a given sensor will be taken, 
which provide at video out 1 a high resolution represen- and the monitor image will be of the same resolution as 
tation of the entire field of view reconstructed from the that of a single element. 

partial views of every sensor, and at video out 2, a The second EPROM 38 contains a table of the coor- 
window image which is a continuous representation of dinates of a pixel in a plane coordinate system with 
a selected portion of the field of view on video output 50 unity vectors along a first line and first row of the sen- 
display 18, as chosen by the cursor 22. sor (further called "local coordinates"). The constants 

Two preferred embodiments are shown in FIGS. 2 addressed by the sweep address generator will then be 
and 3, but those skilled in the art will understand that transmitted to the pre-processors modules. Advanta- 
ge disposition of sensor elements and lenses can be geously, in this EPROM, all sensors elements and lenses 
fitted to the particular application. Each sensor element 55 are identical (in the case of the preferred embodiment) 
is connected by a suitable cable to the electronic inter- otherwise one EPROM per sensor will be needed. The 
face 12. The focal length of the lenses is determined so' EPROM constants will also contain the corrections 
that cvery point of the field of view will have at least needed to compensate the distortion of the optical sys- 
one image point on one sensor. tern. 

FIG. 2 illustrates an arrangement having fifteen sen- 60 , tr . _ _ ~ 

sors for a hemispheric field of view. Each lens system Wmdow Coordinate Rotation Processor 

disposed on the hemispheric-like surface of the optical This module 44 gets its input window parameters 
modular 10 comprises a tight sensitive array formed of through a standard communication port 45. It deter- 
a sensor element 24 and a lens 26. mines the candidate sensor elements for this window 

FIG. 3 illustrates an alternative arrangement having 65 and assigns to every candidate sensor clement one pre- 
eight sensors for ±20* and 360* around field of view. processor 42. Then, it calculates the local coordinates of 
As in the case of the FIG. 2 arrangement, each lens the two vectors formed by the first line and first row of 
system disposed around the periphery of the optical the window in the local coordinate system of each can- 
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didate sensor element, and sends them to all assigned two word parallel signal comprised of line and row 
pre-processors. This is done once per window (once per values of the presently sampled pixel of the sensors 24. 
frame). This procedure allows all pre-processors 42 to The resulting analog output of each sensor 24 is con- 
receive data from one EPROM module, provided all verted to digital data and is directed to the N to 1 multi- 
sensor elements and lenses are identical. If several win- 5 plexer 34, one for the monitor view display 50 and one 
dows are intended for use with one optical sensor, addi- for the window view display 52. 
tional window coordinate rotation processors 44 may ffl . pa^g^ for ^ Monitor View 50 
be needed. 

EPROM 36, addressed by the sweep address genera- 

Pre-processors 42 io tor 30, outputs data which is divided into two words. 

There are P pre-processors 42, where P is defined as The first word contains the identification number of the 

the maximum number of sensor elements having pixels *nsor which has been preselected at the present sweep 

in one window. This number is determined by the dis- and is sent to the multiplexer command register 

position of sensor elements and by the maximum size . 

allowed to "window". In the preferred embodiments " The second word contains the line and tow at dress 

described here, if the maximum aperture angle of the of Ac "fP 1 * 1 P* el m a mom ^r view 50. Line and row 

, , . Z- . u if -L^-i- address data are then sent to the ping-pong frame buffer 

window is twice the aperture angle of one sensor ele- T7. * , . • 1 ? Vi ~.»n; 

4 *u * r *iwT.™™ rsf mn i 48, while corresponding data is received from the multi- 

ment, then P=6 for Aesensor ^gement of FIG 2 ^ output over !he data line 54. 

and p=3 for that of FIG. 3 The role of each pre- P £ x from ^ 

processor 42 ii « fjSST^tS^ censors, the presently filled half of ping-pong frame 
received from the EPROM 38, and to decide whether £ j ^ frame ^ b 

this pixel belongs to the window by comparing these ^ ^ ^ ^ ^ ^ ^ iQT H of 

coordmates wiA the local parameters of window as ^ m framc buffer 48 scans the previously 

received Ifrom the wmdow coordinate rotation proces- ^ ^ ^ ^ ^ providing 

sor 44. The first pre-processor 42 that detects an In monitor on monitor view display 50. 
Window** pixel will send the sensor element number 

and local coordinates of window (i.e., local coordinates Signal Propagation for the Window View 52 

of vectors formed by the first line and first row of win- EPROM 38, addressed by the sweep address genera- 

dpw) to the window processor 40 and inhibit data from ^ t0f ^ outpu ts an address comprised of two words 

other pre-processors. This must be done at video speed which m ^ t coordinates of the sampled pixel in all 

but the functions performed are merely comparisons scnsors> m ^ coordinate system having an axis 

and data transfer. along the first line and first row of sensor. This "Local 

If several windows are required, then there must be P Address" is sent to all pre-processors 42. 

pre-processors dedicated to each window. 35 Meanwhile, the window coordinates rotation proces- 

Window Disnlav Process 40 sor 44 calculates local coordinates and parameters of 

window display rrocess w for ^ ^^date sensor (i.e. the local coordi- 

This module gets as its input the identity of the sensor j^tcs of vectors formed by first line and first row of 
element, the local coordinates of the * *in-window" pixel, window), and sends them to the pre-processors 42 
and the local coordinates of the two vectors formed by 40 w here they are stored for use at next frame, 
the first line and row of the window. Then it performs The pre-processors 42 compare data from EPROM 
the calculation of the coordinates of pixel into window 33 with the date of window received during the previ- 
and sends it to the window ping-pong frame buffer 46. ous frame. If a comparison results in an "in window** 
It also outputs the sensor number of the "in window*' decision, it sends all inputs from EPROM 36 and win- 
pixel to the second N to 1 multiplexer 34; this calcula- 45 d ow coordinate rotation processor 44 to the window 
tion must also be done at video speed. It consists of processor 40 with the addition of the sensor number, 
vector scalar multiplication and scaling. Since there is some overlapping between sensors, it may 
«w » j*«irj ,» tt ~ if. occur that several pre-processors respond together. In 
"Monitor" and "Wmdow" Frame Buffers ^ ^ ^ 8ystcm ^ ^ provided to 

These two modules buffer the data of monitor and 50 install a priority order between pre-processors. 
window image in a ping-pong mode. Each frame buffer Window processor 40 now makes two scalars multi- 
is divided in two similar sections, While one section is plications of: (1) local coordinates of pixel with local 
used to display the previous image, the other section coordinates of the first line of window; (2) local coordi- 
collects data for the next image; this mode of operation nates of pixel with local coordinates of the first row of 
is necessary in the illustrative embodiment shown, since 55 window. These two scalars give the line and row num- 
data is not entered in the same order as the sweeping ber in window. Simultaneously, the sensor number is 
address. sent to multiplexer 34 from window display processor 

Set forth below is a step-by-step description of the 40. 
signal propagation and operation of the preferred em- The window ping-pong frame buffer 46 thus gets line 
bodiment 60 and row address from the window display processor 40 

and data from multiplexer 34, and functions similarly to 
Sensor sampling and EPROM addressing ^ momtor f rame buffer 48, outputting a video signal to 

The sweep address generator 30 produces two differ- the window view display 52. 
ent signals. The first signal is a sweep signal for the N As will be evident in the art, numerous variations 
sensors 24 with the required format to provide a normal 65 may be made in the practice of the invention without 
line by line synchronous scanning of all sensors. departing from its spirit and scope. Therefore, the in- 

The second signal is sent from the sweep address vention should be understood to include all the possible 
generator 30 to the EPROM modules 36 and 38. It is a embodiments and modifications of the illustrated em- 
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bodiments without departing from the principles of the 
invention as set out in the appended claims. 
What is claimed is: 

1. In a system for providing high resolution images of 

a scene, apparatus for simultaneously obtaining a wide 5 
angle image of the scene and a window image of a se- 
lected particular region of the scene, comprising: 

an optical module having a plurality of light sensitive 
arrays disposed therearound, each light sensitive 
array comprising a sensor element and a lens ele- Ju 
ment for providing electrical output signals repre- 
sentative of the scene; 

means for connecting the output of each light sensi- 
tive array to an electronic interface, said electronic 
interface comprising means for processing the sig- 15 
nals from said light sensitive arrays; 

monitor view display means; 

window view display means; and 

means for connecting the output signal from said 
electronic interface to said monitor view display 
and said window view display to provide on the 
monitor view display a high resolution image of the 
entire scene obtained by the combination of a plu- 
rality of images acquired simultaneously on a plu- 25 
rality of sensor elements and on the window view 
display a high resolution image of a selected partic- 
ular region of the scene obtained by the integration 
of data received from the sensor elements. 

2. In a system for providing high resolution images of 3Q 
a scene as set forth in claim 1 wherein said optical mod- 
ule comprises a hemispheric-like surface having light 
sensitive arrays disposed around said surface. 

3. In a system for providing high resolution images of 

a scene as set forth in claim 1 wherein said optical mod- 35 
ule comprises a polygonal shaped number having light 
sensitive arrays disposed around the periphery of said 
member. 

4. In a system for providing high resolution images of 

a scene as set forth in claim 1 wherein said electronic 40 
interface comprises means to sample all sensor elements 
in the optical module, means to collect all pixel data of 
a determined field of view on said sensor elements, and 
means to provide to the monitor view display signals 
corresponding to an image of the entire scene, and to 45 
provide to the window view display signals corre- 
sponding to an image of a selected particular region of 
the entire scene such that both images can be viewed 
simultaneously. 

5. In a system for providing for high resolution in> 30 
ages of a scene as set forth in claim 4 wherein said elec- 
tronic interface further comprises means to change the 
size and position of the window image on the window 
view display means. 

6. In a system for providing high resolution images of 55 
a scene, a method for simultaneously obtaining a wide 
angle image of the scene and a window image of a se : 
lected particular region of the scene, comprising the 
steps of: 

obtaining from an optical module having a plurality 60 
of light sensitive arrays disposed therearound, elec- 
trical output signals representative of the scene; 

connecting the signals from each light sensitive array 
to an electronic interface; 

processing the signals at the interface to provide two 65 
sets of output signals respectively representing the 
entire scene and a selected window region of the 
entire scene; and 
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connecting the output signals from said electronic 
interface to a monitor view display and a window 
view display to provide on the monitor view dis- 
play a high resolution image of the entire scene 
obtained by the combination of a plurality of im- 
ages acquired simultaneously on a plurality of light 
sensitive arrays and on the window view display a 
high resolution image of a selected particular re- 
gion of the scene obtained by the integration of 
data received from the light sensitive arrays. 

7. In a system for providing high resolution images of 
a scene as set forth in claim 6 further comprising the 
steps of sampling sensor elements in the optical module, 
collecting all pixel data of a determined Meld of view on 
said sensor elements, providing to the monitor view 
display signals corresponding to an image of the entire 
scene, and providing to the window view display sig- 
nals corresponding to an image of a selected particular 
region of the entire scene such that both images can be 
viewed simultaneously. 

8. In a system for providing for high resolution im- 
ages of a scene as set forth in claim 7 further comprising 
the step of changing the size and position of the window 
image on the window view display means. 

9. In a system for providing high resolution images of 
a scene scanned by a plurality of sensors, apparatus for 
simultaneously providing a monitor image of the scene 
and a window image of a selected particular region of 
the scene comprising: 

means for receiving input signals from an arbitrary 
number N of sensors, each sensor scanning a por- 
tion of an image of a scene, said N sensors covering 
the entire scene with possible overlapping, 

means for receiving a set of numerical input signals 
through an electronic interface, said numerical 
input signals representing the position, form and 
size of an arbitrary number of arbitrary window 
portions of the image of the scene, and 

means for providing output signals, after each set of 
N input signals has been received from the N sen- 
sors, representing the following data: 

(a) a monitor address number representing the posi- 
tion of one pixel in a monitor image frame in- 
tended to represent the entire scene, 

(b) window address numbers, equal to or greater 
than the number of windows, each window ad- 
dress number representing the position of one 
pixel in a window image frame intended to repre- 
sent one portion of the scene defined by one set 
of said numerical input signals, 

(c) a monitor selection number defining the sensor 
from which data will be taken, representing the 
light intensity of the point in the image of the entire 
scene at the position given by said monitor address 
number, and 

(d) window selection numbers, equal to or greater 
than the number of windows, defining the sensor 
from which data will be taken, representing the 
light intensities of the points in each window 
portion of the image of the scene at the position 
given by said address number. 
10. In a system in accordance with claim 9 further 
comprising frame buffer means for collecting said out- 
put signals from the system and for displaying at an 
external display device the received data and forming 
an electronic image of the monitor view and the win- 
dow view. 
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11. In a system in accordance with claim 10 further 
comprising means for connecting the N sensors, means 
for connecting to an external electronic device capable 
of providing numerical output signals defining the size, 
form and position of a portion of the window image of 
the scene, means for switching and connecting the sen- 
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sors selected by said monitor and window selection 
number output signals to said frame buffer means and 
means for connecting external display devices capable 
of providing monitor and window images of the entire 
scene and of the selected portions of the scene. 
***** 
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